Persist full job completion batches concurrently - #1434

DRAFT: not ready for review. Opened to evaluate on its own merits and schedule separately.

The batch completer persists finished jobs with one JobSetStateIfRunningMany query at a time. That works well under light load, but under sustained heavy load it becomes the bottleneck. While one 5,000-job batch is being written, workers keep finishing jobs and a second full batch piles up behind it. When the backlog reaches its limit, completions start waiting on the completer, so workers sit idle even though the database could easily run another query.

Here's an example: a client works down a large queue of short jobs. Each completion query takes a few milliseconds, and in that time workers finish thousands more jobs. The next batch is already full by the time the current query returns, so the completer is always behind, and job throughput is capped by the latency of one query at a time.

With this change, the completer can run up to two completion queries at once, each still capped at 5,000 jobs. The second query starts only after its own backlog has filled a batch, so light and sparse traffic keeps coalescing exactly as it does today and doesn't use an extra connection. Concurrency is opt-in through two new optional interfaces, riverdriver.ExecutorJobCompletionConcurrency and riverpilot.PilotJobCompletionConcurrency. The pgx and database/sql pool executors and the standard pilot report two. Transaction executors report one, since one transaction can't run statements concurrently. SQLite and any pilot that doesn't implement the interface stay serial.

Running two batches at once introduces a hazard. Two batches could hold different executions of the same job, and PostgreSQL might lock their rows out of order, which would let an older result overwrite a newer attempt. To prevent that, the completer tracks in-flight and retrying job IDs. A later result for the same job waits until the earlier batch finishes, while batches for different jobs still persist concurrently. On shutdown, the completer waits for active queries and then makes one bounded final flush attempt, as before.

There are two user-visible side effects, both noted in the changelog. Subscribers may get completion events for different jobs in a different order than before, though events for the same job stay in order. And the completer may briefly use one extra database connection.

I compared river bench on master (6cdaf38) and on this branch, with runs alternating between the two builds against the same local database:

The -n mode inserts all the jobs up front and then measures how fast they're worked down, which is the backlog case this change targets. The --duration mode inserts and works at the same time, and there the inserter limits throughput, so the gain is small. These numbers come from one laptop (Apple Silicon M4, 14 cores, Homebrew PostgreSQL 18.6 on localhost, Go 1.27.1). Absolute numbers will differ on other machines and on a networked database, where query latency is higher and overlapping queries may help more.

This also brings Go's completion throughput back in line with River's Rust implementation, which persists completion batches the same way.

The first commit adds the capability interfaces and a driver test that runs the advertised number of disjoint completion batches concurrently through a pool executor. The second commit changes the completer, with tests for capability negotiation, a sparse second batch that stays queued until it's full, and a job whose results are forced to interleave across concurrent batches.